iCLEF 2001 at Maryland: Comparing Word-for-Word Gloss and MT

For the rst interactive Cross-Language Evaluation Forum, the Maryland team focused on comparison of term-for-term gloss translation with full machine translation for the document selection task. The results show that (1) searchers are able to make relevance judgments with translations from either approach, and (2) the machine translation system achieved better e ectiveness than the gloss translation strategy that we tried, although the di erence is not statistically signi cant. It was noted that the \somewhat relevant" category was used di erently by searchers presented with gloss translations than with machine translations, and some reasons for that di erence are suggested. Finally, the results suggest that the F measure used in this evaluation is better suited for use with topics that have many known relevant documents than those with few.